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[57] ABSTRACT 

A universal command generator that generates commands 
that are recognizable by multiple contexts, where the uni- 
versal command libraries are loaded automatically and 
dynamically as the user switches from context to context. 
Additionally, a novel gesture recognition method is provided 
to allow the user to define symbols specific to the users 
preferences that will work universally. The preferred 
embodiment of the present invention comprises novel meth- 
ods and apparatus that provide this functionality such as: a 
method for translating signals from an input device into 
commands operating on a context-sensitive basis, a method 
for recognizing and translating a command received from an 
input device into a command recognizable by a second 
context, and a device for interpreting and translating com- 
mands entered into a first device into commands that are 
understandable by a second device. 
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Symbol Editor - Step 1 of 2 



Vector view: 




Create your new symbol by clicking 
and dragging the mouse. 

Click twice on a point to stop 
capture. 

You can modify the shape by 
clicking and dragging the black 
handles. 



The data you provide for this page 
will be used for recognition. 
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Example Sequence: 

"acabddc" 
0123456 - Element Indexes 

Possible Swaps: 

"ac" 
"ca" 
"bd" 



Application of Bit Mask: 


Results of Swappincr: 




acabddc 
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00 0 
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00 1 
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SYSTEM, METHOD, AND APPARATUS FOR There is a need for a universal command generator that uses 

GENERATION AND RECOGNIZING input-friendly input commands, such as gesture recognition, 

UNIVERSAL COMMANDS which will allow users to create their own symbols or letters 

to represent universal commands, and yet still be highly 

BACKGROUND OF THE INVENTION 5 accura te in recognizing the gestures. There is a need for this 

1. Field of the Invention gesture recognition system to allow any symbol to be used 

The present invention relates generally to methods for as aD in P m > and tnere » a need for the svstem t0 °P erale wilh 

generating universal commands, in particular, to a system, conventional computer systems, without any special 

method and apparatus for creating user-defined commands modifications, such as adding a cursor input device, 

that are understandable by a second device or application 10 Additionally, there is a need to be able to operate existing 

that otherwise would not recognize the command issued by technological devices, such as a television using gesture 

the user. Additionally, the present invention employs novel recognition without altering the device itself, 

gesture recognition technology to recognize the commands SUMMARY OF THE INVENTION 
by the user. 

2. Description of the Related Art 15 Ia accordance with the present invention, a universal 

^ . . . j t . t t command generator is disclosed. The universal command 

Computer systems today are essential in the workplace & . . . . t , 

. A \_ , T c ' . , , f 4 . , generator overcomes the problems with the prior art by 

and at home. Users of a typical computer use multiple ° . . — , n~-i— t— — =i— — 

4 . . \ r\r. i*- i using an improved method or converting a user input into-j 

applications on their computer. Often, users run multiple , ui— u — i — * ^ • 

rr .. ... i . * i- *• commands that are recognizable by an application or device 

applications in the same work session. Applications nn ^ «_ j— - — i- — * u . x. . - - . - -„ ~ 

, . , - Ci , , ^ A . , rr , 20 that ordinarily would not be able to interpret the user 

designed for users often have shortcuts or quick command ' ^-^-j- _/ _ . _ „ # . , , , . 

, & . . , . . . 4 *i. • i- *■ i- -command) The converter is fully customizable by the user 

keys designed to make it easier to use their application. For £ . 4 • • . L _j? iL 

, , . . Li »l sucn that a given input can be made to perform the same 

example, a word processing program might enable the . to c iL \ . 

- , a , . „. \ i* • * j function, regardless of the application that is active or the 

Ctrl-S combination to result in a save command. Or a spread 1 . & 4 , . f r 0 . , iL 

, A i , . * . c j * ' operating system being used. Similarly, the user can cus- 

sheet program may use a Ctrl-C to cut a block of data in „ c . r . ~ . * t . t ■< .„ . t , . t 

, f i j ™ . • . . 25 tomize the same input so that it will initiate different, but 

order to be moved. These commands are very convenient to - „ , a j j r *i_ * * r 

- ,. r • • r> nilly definable actions, depend of on the context or apph- 

use, and typically after using an application for a period of ' ™. . . ' . i , 

4 . ' L ^ r , 3 iL , ~ 4 rr r cation. The present invention employs novel gesture recog- 

time the user learns the short cuts. t . r . , / ■ * u *u 

. mtion technology to recognize gesture inputs by the user, 

Unfortunately, all applications do not use the same short- ^ providing tne ease of gesture inputs to every application 

cuts for the same functionality. Applications are designed to 30 and deyice The gestUfe recognition technology is not lim- 

perform different functions, are written by different ited to a single sel of characlers> and yet is still highly 

companies, and in general do not follow a universal standard accurate . Gestures can be symbols, characters, or any ere- 

for quick command definitions. This forces the user to atior) of the usef ^ input device can be any pointing 

memonze or look up a different set of commands each time devicC) a m0US6j graphic tablets> touchpad or 

the user wishes to use a different application. When a user 35 (ouch SCfeen) or any deyice that can CQntrol me position of 

is switching rapidly between several contexts, this can a cursor> visible or not ^ present invention also allows a 

become very burdensome and confusing. Also the user is rcmotc dcvice> such ^ a teievisiorl| t0 bc controlled by 

generally limited to the quick commands defined by the use r-defined gestures. The preferred embodiment of the 

application. Often their quick commands are not intuitive for present invention com prises novel methods and apparatus 

the user. For example, the creator of the program may 40 that provide this functionality such as: a method fonftrans^ 

believe that one quick command, for example ctrl-c, is easy ^ .^.^ from - an - input d evi ce rdnto^mWnds"operating^ 

to remember as copy, while a user may believe that ctrl-c, on^oontext^Sifiw bi^a-mothod -for recognizing and A 

due to past experience is "cut" for the specific context of tra^lating a-cmnl^ w 

used. In the prior art, however, the user would be forced to com^naii^recogmzable by a second contextrand a devjee for-^ 

relearn ctrl-c as cut. Also, the prior art quick commands are 45 jntgrp re tmg-and-teans^^ 

generally limited by keyboard inputs. These may not be device-into-com!^^ 

descriptive enough for a user to remember. Cde^ife^ 

There are more user friendly inputs, however, such as the 

technique of gesture recognition. Using gesture recognition, BRIEF DESCRIPTION OF THE DRAWINGS 

a user can draw a character on a screen usually being 50 Fia x is a schematic of a pre f erre d embodiment of the 

restricted to using a pen-based system and the computer uniyersal command erator stem in accordancc with thc 
system will recognize the character. Often, in the prior art, 
the user is forced to learn a pre-defined alphabet in order for 

the system to recognize the character. Thus, prior art gesture , . , , 

recognition techniques would not prove helpful in allowing 55 ^ 0mmand S eneraUon ln accordance vatb the present mven- 

a user to define a command or create a command that is lon * 

easily remembered. Finally, a user may wish to employ an FIG * 3 * a fl° wc hart of the preferred method of recog- 

existing device, such as a television, with gesture recogni- mzin S the current context in accordance with the system of 

tion technology that will allow the user to input symbols to lhe P resent invention. 

represent commands. However, the user may want to use 60 FIG - 40 & a flowchart of the preferred method of recog- 

gesture recognition without having to purchase a new tele- nizing a universal command in accordance with the system 

vision. of the present invention. 

Therefore, there is a need for a system that can generate FIG. 4b is a flowchart of the preferred method of deter- 

universal commands that will work on multiple applications, mining if a gesture is being drawn in accordance with the 

operating systems, and devices. There is a need for this 65 present invention. 

system to allow the user to define their own commands and FIG. 4c is a flowchart of the preferred method of recog- 

still have them be recognized by the device or application. nizing a gesture in accordance with the present invention. 



present invention. 

FIG. 2 is a overview flowchart of the method for universal 
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FIG. 4d is a graphical representation of a script letter interact directly with the CPU 108 by inputting commands, 

being segmented in accordance with the present invention. such as save file, or minimize window, display available 

FIG. 4e is a flowchart of one method of performing memory, etc. Applications 132 can be any conventional 

dynamic tolerance in accordance with the present invention. applications such as Word (TM), WordPerfect (TM), Corel 

™^ AC . * j i_ j * • i. i - 5 Draw (TM), Lotus 1,2,3 (TM), and can be located in 

FIG. 4/ is a preferred method of executing hotkeys in memory 152 Ty pically a ppi icat ions are coupled to the 

accordance with the present invention. operating system, which translates the application instruc- 

FIG. 5 is a flowchart of the preferred method of translat- tions into instructions for the CPU 108. Applications can 

ing a universal command in accordance with the present also be directly coupled to the CPU 108 through the databus 

invention. jQ i4g. Applications can be operated by the user 100 by issuing 

FIG. 6a is a flowchart of the preferred method of creating application-defined commands such as cut, paste, copy, 

a universal command in accordance with the present inven- draw, etc. 

lion. Preferably, the memory 144 stores modules containing 

FIG. 6b is a screenshot of a preferred user interface for libraries used by the system of the present invention. The 

selecting commands to be assigned to a universal command libraries are preferably contained in separate memory 

file in accordance with the present invention. modules, and are coupled to the data bus 148. Memory 

FIG. 6c is a flowchart of the preferred method of selecting module 116 contains libraries of universal command files for 

a symbol or hotkey to represent a universal command in tnc applications stored in memory 152, Memory module 120 

accordance with the present invention. contains a library of default universal commands. Memory 

FIG. la is a flowchart of the preferred method of render- 20 module 124 contains libraries of universal command files for 

ing a gesture in accordance with the present invention. the grating systems 136 being used by the CPU 108. 

FIG. lb is a flowchart of the preferred method of defining ™ tmox ? m ° dule 126 ^tains a library of symbols used by 

, , . . . l *t j ■ i j * the user. Memory module 128 contains routines whose 

a relationship between a hotkey and a universal command in - A . .. L *. ., , . r^^o ^ * j n L 

_ „ t , functionality is described m FIGS. 2-5, and will be 

accordance with the present invention. , ■ ■ i rr . , , ... ^ r 

r ^„. r , r r , . „ .75 described below. Universal command libraries 116, 124 

FIG. 7c is a screenshot of a preferred user interface in slQre uniyersal command files t0 each application 

accordance with the present invention. or operating systerDi Each umver sal command file has a 
FIG. 8 is a flowchart of the preferred method of deleting gesture and/or a hotkey sequence contained within it. The 
a universal command in accordance with the present inven- sequence assigned to the universal command file is unique 
tl0r1, 30 to its library. Each file also has one or more entries assigned 
FIG. 9 is a schematic of a hardware embodiment in to it. An entry in a universal command includes any corn- 
accordance with the present invention. mand used by an application or operating system, a text 

string the user wants displayed, a sound file that the user 
desires to hear, or any combination of the above such as a 

35 macro. Those skilled in the art would know that other types 

FIG. 1 depicts a preferred embodiment of the universal of commands or entries could also be used as universal 

command generation system 10 of the present invention. commands. An entry in a universal command file may also 

The universal command generation system 10 preferably include a command recognized by a second device 140, such 

comprises an input device 104, a display device 112, a as increase volume, skip disc, etc. Each universal command 

permanent storage device 156, a central processing unit 108, 40 can have as many entries as the user 100 desires. The entries 

a first memory 144, and a second memory or object device can be stored in whatever order the user 100 desires. The 

152. Those elements are preferably coupled to a common library of independent symbols 126 contains all the symbols 

bus 148 in a Von Neuman architecture. A user 100 uses an the user may draw and assign to the library 126. Once in the 

input device 104 to communicate with an object device 152, library 126, the symbols are available to be assigned to a 

The object device 152 can include an application 132 45 universal command created by the user 100. The library of 

resident in memory 152 of a computer, an operating system default universal commands 120 is described in further 

136 resident in memory on a computer 136, or a second detail below. 

device 140 such as a television, a CD player, or a stereo FIG. 2 shows an overview of the operation of the pre- 

system. Hie second device 140 can be physically coupled to ferred embodiment of the present invention. In step 201, the 

the databus 148, or it can be remotely coupled via infrared, 50 user 100 uses the input device 104 to send an input signal to 

or another conventional radio frequency communication the object device 152. In the prior art, the user would have 

method. to remember the correct input signal to perform the desired 

The object device 152 can be one object or multiple command for the specific context the user was working 

objects. The input device 104 can be any conventional input within. For example, in a computer system running multiple 

device, such as a mouse, keypad, keyboard, touchpad, 55 applications, the user would need to remember that ctrl-s is 

graphics tablet, touch screen, or any input device capable of save in a word processing application, but that ctrl-s may be 

controlling the position of a cursor. The input device 104 is scroll in a web browsing application, 

coupled to a data bus 148. The data bus 148 is coupled to a In accordance with the present invention, the system 10 

central processing unit (CPU) 108. The CPU 108 may be any dynamically recognizes the current context 200 of the input 

one of the conventional processors on a microcontroller, for 60 device 104. This is important for systems having multiple 

example, manufactured by Intel or Motorola. The CPU 108 contexts. A context includes an application 132, an operating 

is coupled to a set of memory modules 144, permanent system 136, or a second device 140. However, if the system 

storage 156, and a display 112. The operating system 136 has only one context, then the system may not need this 

can be any conventional operating system, such as a Win- functionality. After recognizing the current context, the 

dows (TM), Unix (TM), or OS/2 (TM) operating system. An 65 system recognizes a universal command 204 entered by the 

operating system 136 is coupled to the CPU 108 through a user 100. If the input signal is not a universal command, the 

databus 148, and provides an interface for the user 100 to input signal is routed to the current context for processing. 



DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENT 
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In the preferred embodiment, the universal command is 
preferably entered as a gesture. A gesture can be any 
character, symbol, or other creation of the user. However, the 
universal commands may also be keystrokes or combina- 
tions of keystrokes. 

After recognizing the universal command 204, in the 
preferred embodiment the system translates 208 the user's 
universal command into a command, commands, text 
strings, or other information recognized by the current 
context. The translated universal command is sent (or string 
of commands and data produced by the translation) to the 
current context to generate the desired output 212. 

FIG. 3 depicts a flow chart showing more detail of a 
preferred method for recognizing 200 the current context 
and dynamically and automatically loading the correspond- 
ing universal command libraries. In the present invention, 
the user may have several contexts stored in the memory 152 
of the system 10. When the user moves the input device 104 
over an active context, the system 10 recognizes the context 
and loads the library of universal commands corresponding 
to that context into memory. More specifically, in the pre- 
ferred embodiment, the system 10 first determines 304 the 
coordinates of the input device 104. These coordinates are 
generally x-y coordinates showing the location of the input 
device 104 in relation to the active area of display or they 
could be the cursor location if the keyboard is providing the 
input. In a Windows environment, for example, the system 
10 can constantly query the Windows operating system for 
the location of the input device 104, in this case, a mouse - 
type controller 104. In a second embodiment, the system 10 
can identify the current context by asking the operating 
system which context is currently active. This can be a 
user-defined embodiment, allowing the user to select 
whether the system 10 operates dynamically or only upon 
activating a context. 

After determining the input device 104 location, the 
system 10 preferably identifies the current context by deter- 
mining what context the input device 104 is located over 
308. In a system 10 of multiple contexts, the input device 
104 will always be over one context, for example, an 
operating system context or ones of the applications having 
a window on the display device 112. Once the current 
context is determined, the system 10 then retrieves infor- 
mation 312 concerning the current context, such as the 
identification of the context. In a Windows environment, the 
contexts are displayed in windows, and the system 10 can 
use the location information of the input device 104 to query 
the operating system regarding what context window cur- 
rently overlapped by the input device 104 location. The 
operating system 136 will return the handle of the window, 
which will allow the system 10 to retrieve whether the 
current context is a child window of a parent application. If 
the current context is a child window, then the system 10 will 
find the top-level parent window of the child window. Once 
the handle of the top-level parent window is retrieved, the 
operating system is asked for the name of the application, 
the class of that parent window, and will thus recognize the 
current context. 

After identifying the current context in step 308, the 
system 10 then checks to see if the universal command 
library of that context is loaded 320 in the memory 116, 124. 
Each application 132 or operating system 136 will have a 
universal command library memory module 116, 124 asso- 
ciated with it. This library will contain the universal com- 
mand files used by that application or operating system and 
information about the corresponding context, for recogni- 
tion. If the library is not loaded, the system 10 will load the 
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corresponding library 328. However, if there are no libraries 
assigned to a context, a set of default universal commands 
are loaded 332. These default gestures can be any commonly 
used gestures such as S for save, D for delete, P for print, or 
symbols such as arrows or curves. The user can later modify 
the default universal command library to suit his or her own 
needs. In one embodiment, hotkeys can be used to represent 
universal commands. In that case, the system 10 instructs 
330 the operating system to ignore any hotkeys stored in the 
universal command library of the current context and thus 
overlaying the information entered with the keyboard, 
accessing it before all other applications. After loading the 
corresponding library, or a default library, the system 10 is 
ready to interpret universal commands. 

As shown in FIG. 4a, in the preferred embodiment, the 
system 10 distinguishes between universal commands that 
are keyboard entries and universal commands that are ges- 
tures. After determining that a gesture is being drawn 404, 
the system 10 then executes the Gesture Execution module 
408. If the system 10 determines that a hotkey has been 
input, then the hotkey execution module 414 is called. If the 
system 10 determines that no hotkey has been input, then the 
method moves to step 212 and output is generated by the 
current context without processing in accordance with the 
present invention. 

FIG. 4b shows more detail of a preferred embodiment of 
determining if a gesture is being drawn, and if so, drawing 
the gesture on a display 112. First, the system 10 determines 
whether an input device 104 is being configured for gesture 
drawing 420. In one embodiment, if an input device 104 is 
a mouse, then the system 10 would determine if a right 
mouse button is being pushed. Those skilled in the art will 
realize that a variety of other combination of keyboard and 
mouse inputs could also be used. If an input device 104 is 
being configured for drawing a gesture, the system 10 draws 
a dot 424 at the current location of the input device 104. The 
system 10 then checks if the device is still configured for 
drawing a gesture 428. If it is, then it draws another dot at 
the current location of the input device 104. In a preferred 
embodiment, the user 100 must depress the right mouse 
button and move the input device 104 to draw a gesture. 
When the input device 104 is no longer configured for 
drawing a gesture, the user has completed the gesture The 
gesture can be shown on the display 112 as it is drawn. When 
the user is done drawing, the symbol will be deleted, and the 
gesture execution module 408 is called. 

In the preferred embodiment, the system 10 captures all 
input signals prior to being processed by any other compo- 
nent of the computer system. This avoids any conflict 
between an application's or operating system's definitions of 
input device configuration and the system 10 of the present 
invention's definitions of input device configuration. For 
example, in the Windows (TM) environment, if the right 
mouse button is used as the gesture drawing configuration, 
there might be a conflict between gesture drawing and the 
Windows definition of the right mouse button creating a 
pop-up menu. However, the system 10 in the preferred 
embodiment will receive the input device configuration prior 
to the operating system. It will simulate the operating system 
command if it determines that the user intended to create a 
pop-up menu. Otherwise, it will allow a gesture to be drawn. 

At this point, the system 10 is ready to recognize the 
gesture representing the universal command 408. FIG. 4c 
shows a preferred embodiment of gesture recognition in 
accordance with the present invention. Preferably, the first 
step is to delete the dots of the drawn gesture 458. Then, the 
system 10 will locate the boundaries 436 of the drawn 
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gesture. Id the preferred embodiment, this is accomplished 
by placing a bounding box around the gesture. Next, the box 
is subdivided 440 into blocks by creating multiple axes. In 
a preferred embodiment, two vertical axes are drawn equi- 
distant from the left and right side of the bounding box and 
each other. Two horizontal axes are drawn equidistant from 
the top and bottom of the box and each other. Each axis has 
a label. In one embodiment, the two horizontal axes 
(including the perimeter axes) are labeled a and b. The two 
vertical axes are labeled c and d. Then the sequence in which 
the input signal intersects the axes is determined. For 
example, as shown in FIG. 4d, if the user drew a "P" by 
drawing a vertical stroke and then a semi-circular stroke, the 
sequence, for a perfectly drawn P. would be "bacdadc", 
assuming the user began drawing the P at its bottom. 

In a preferred embodiment in accordance with the present 
invention, the system 10 also detects straight lines. Straight 
lines are important to detect separately because users 100 
rarely draw lines straight. Therefore, the system 10 can 
separately recognize a straight line. After the boundary box 
is created around the gesture, the height to width ratio of the 
box is calculated. If the box height to width ratio is less than 
a fixed value for a vertical line, or greater than a fixed value 
tor a horizontal line, it is assumed that the user intended to 
draw a straight line. The fixed value is a predefined width to 
height ratio, such as 1 to 4. No axis intersection calculations 
are needed; instead, the system 10 will search for a universal 
command file stored in the universal command library of the 
current context that is also labeled a straight line, and 
execute that command in accordance with the method 
described below. 

Additionally, in another embodiment, the system 10 can 
distinguish between a straight line and a line that was drawn 
on top of itself. After determining that a gesture is a straight 
line, the system 10 will compare the startpoint of the gesture 
to the endpoint of the gesture. If they are located within the 
same region, then the system 10 determines that the gesture 
was a line drawn on top of itself. This gesture may have 
special meaning for the user, and the system 10 searches the 
universal command files for a file tagged as a the line drawn 
on top of itself gesture, and sends the entries associated with 
that universal command file. 

The system 10, after determining the sequence the input 
gesture intersects the axes, then compares the determined 
sequence with the stored sequences in the universal com- 
mand files assigned to the universal command library of the 
current context 448 which has been automatically and 
dynamically loaded into memory, as described above. In a 
system 10 with one context, the one library would always be 
loaded into memory. There is one universal command file 
for each universal command. Each universal command file 
has a unique gesture sequence assigned to it. This system 10 
then determines whether any stored sequences match the 
determined sequence. If one does, then the dots drawn on the 
display are deleted 458, and the universal command file is 
translated 208. The gesture recognition technology is very 
tolerant. The second P looks dissimilar to the first P, yet it 
will still be immediately recognized because its sequence is 
also "bacdadc." 

However, if the system 10 does not find a match, then it 
performs dynamic tolerance 457. Dynamic tolerance is 
preferably performed as described in FIG. 4e. The sequence 
the gesture intersects the axes is a sequence of elements in 
a string, such as "acabddc" If the drawn gesture does not 
exactly match a stored perfect gesture, then the gesture is 
modified until a perfect match is found. The misdrawn 
gesture is transformed, by changing its shape, until its shape 
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matches the shape of the stored gesture it was meant to be. 
The system 10 accomplishes this by repeatedly swapping the 
positions of the elements in the determined sequence until a 
match is found, no match is found, or a pre-defined swap 

s limit is met. The swapping process begins by finding the 
indexes of the elements of the sequence that can be swapped. 
Similarly oriented axis intersections will not be swapped. 
Thus, only those elements next to an element of opposite 
orientation are indexed to be swapped. For example, as 

jo shown in FIG. 4e, for a string "acabddc", the only possible 
swap pairs are a-c, c-a, and b-d. Once the elements that can 
be swapped are determined, the system 10 calculates the 
number of possible combinations of swaps that can be made. 
The number of combinations are calculated by calculating 2 

35 to the power of x, where x is the number of possible swaps. 
The base 2 is used because there are only two possibilities 
for swapping: swap, or no swap. For example, as in FIG. 4e, 
if there are three possible swaps, then there are 8 possible 
combinations of the elements in that sequence. The system 

20 10 then sets a counter from one to the number of possible 
combinations. In our example, the counter would go from 0 
to 7. Next, the system uses the binary representation of the 
counter value and applies it using a bitmask to each indexed 
element in the sequence. Only the indexed elements that can 

25 be swapped that are also marked by a 1 will be swapped. 
Sometimes, as shown in FIG. 4e, there may be multiple 
swaps to be performed in one counter loop. One skilled in 
the art would realize that modifications can be made to this 
system without altering its functionality, such as having the 

30 counter count backwards, or swapping in a different order. 
After a swap is made in one counter loop, the new 
swapped sequence is compared to the stored sequences of 
the universal command files of the universal command 
library of the current context. If there is a match, then the 

35 universal command is translated 208, If there is no match, 
then the system 10 asks whether a preset swapping limit has 
been met 474. If it has, then the message, "No Command 
Performed'' is displayed 476, and the system 10 returns to 
recognizing the current context 200, If the limit has not been 

40 met, then the counter is increased, and the processed is 
repeated until either all of the possible combinations have 
been compared or the preset limit is reached. The user can 
define how tolerant the system 10 is to misdrawn gestures by 
setting the number of swaps that can be performed. The 

45 more swaps that are allowed, the more tolerant the system 10 
is. The swapping method in accordance with the present 
invention provides for highly accurate and flexible gesture 
recognition without limiting the user to a single alphabet. 
Since the user can create gestures to represent universal 

so commands, as described below, the user can create whatever 
symbol the user desires to represent a command and still 
have it recognized by the object device 152. Thus, the user 
can create symbols that are easy to remember for the user, 
and apply them in every context. 

ss If the universal command is determined to be a hotkey 
412, as shown in FIG. 4a, then the hotkey execution module 
414 is called. In the preferred embodiment, hotkeys can be 
used as universal commands. When a hotkey is entered, the 
hotkey is processed in accordance with FIG. 4/. The code 

60 representing the keyboard input is determined 480. The 
system 10 compares the determined code 484 to the stored 
hotkey codes in the universal command files of the universal 
library of the current context. If no matching sequence is 
found 488, a "No Command Performed" flag is raised, and 

65 the system 10 returns to recognizing the current context 200. 
If a match is found, then the system 10 moves to the translate 
universal command module 208. 
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FIG. 5 shows a preferred embodiment of the translate 
universal command module 208. Each universal command 
file may have several entries. These entries can be 
commands, such as save, or cut, or delete, or they can be text 
strings, menu commands, sound files, launching an appli- 
cation or any other event performed by an application 132 or 
operating system 136. The command can also be change 
channel, skip disc, or any other command understandable by 
a second device 140. After recognizing the universal 
command, and finding the corresponding universal com- 
mand file, the system 10 preferably determines if there are 
any entries 504 in the universal command file. If there are no 
entries, then the system 10 displays a "Command Not 
Defined" message 508 and returns to step 200. If there is an 
entry, the system 10 sends the command or text string to the 
current context for execution 512. In this embodiment, the 
system 10 then determines whether there are other entries 
516 in the universal command file. If there are, it returns to 
step 512 and sends those entries to the current context for 
execution 520. After all the commands have been sent, the 
system returns to step 200. The current context generates the 
requested output 212 after receiving the commands. 

FIG. 6a shows a system 10 for generating universal 
commands, in accordance with a preferred embodiment of 
the present invention. The user 100 activates the universal 
command editor 602. This may be enabled by a right mouse 
double click in a Windows (TM) environment, or by any 
other means of starting a program. A menu can be displayed 
here, displaying options for the user to select between. The 
user 100 will select the "Create a Universal Command" 604 
option. The user 100 then selects 606 or specifies the library 
of universal commands to be edited. At this point, in the 
preferred embodiment, the user can select an existing ges- 
ture from a library of symbols to represent the new universal 
command, the user may draw a new symbol to represent a 
universal command, or the user may select a Hotkey to 
represent a universal command 608. After selecting an 
option, the system 10 then displays the options 612 of 
commands or text that can be assigned to the universal 
command. A screen shot is shown in FIG. 6b of a preferred 
embodiment of the user interface used to display options. 

In one embodiment, the user 100 will select the command 
or text or menu function the user 100 desires to execute first 
616, and the choice is assigned 620 to the universal com- 
mand file 620. The interface is updated to reflect the choice 
made 622, and the user is allowed to add more commands 
or text to the universal command 624. The order in which the 
user assigns commands is the order in which the commands 
will be executed, as described above. The user can change 
the order of or even delete the commands at any time. After 
all of the commands or text are assigned to the universal 
command file, the universal command file is stored and 
assigned to the universal command library 628 for the 
application to be edited. After assigning the command to the 
appropriate library, the universal command editor is exited 
632. The user can use this feature to assign the same gesture 
to perform a single command in all applications. For 
example, if a user wanted an S to mean save for every 
application, the user can simply create the S gesture as a 
universal command to save for one application. Then, the 
user could create the save universal command for every 
application even though the original access to that command 
is different in several applications and assign the S symbol 
to it. Afterwards, the user will only have to remember that 
S means save, regardless of what specific context the user is 
in. 

FIG. 6c shows more detail of a preferred embodiment of 
the user selecting to draw a gesture, use an existing gesture, 
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or use a hotkey as the universal command 608. First, the 
default gestures and hotkeys from the default universal 
command library are retrieved 636 and loaded into memory. 
Next, the gestures and hotkeys being used in the application 

s to be edited are retrieved 640. They are compared 642 to 
eliminate any default gestures or hotkeys in use by the 
application to be edited. Then, the unused default gestures 
and hotkeys are displayed, and separately the gestures and 
hotkeys used by the application to be edited are displayed 

10 644. The user 100 can now see what has been used and what 
is available. Of course, the user need not use a default 
gesture or hotkey, but can create a totally new gesture or 
hotkey if the user chooses. 

The user 100 is now shown a choice of drawing a gesture, 

15 selecting an unused default gesture, and defining a hotkey to 
represent the universal command 648. If the user chooses 
652 to draw a gesture, then a new universal command file is 
created to store the information 656, and the draw a gesture 
routine 700 is called. If the user chooses to select an existing 

20 gesture 664, then a new universal command file is created to 
store the information 656, the gesture selected is displayed 
660, and the user is given an option to edit the gesture 662. 
If a user 100 selects to create a hotkey 664, then a new 
universal command file is created to store the information 

25 656, and the create a hotkey routine is called 668. 

In one embodiment of the present invention, the user 100 
can simply draw a symbol independently of a universal 
command. After doing so, the drawn symbol is stored in a 
library of symbols assigned to the current context. When the 

30 user wants to create a universal command in that context at 
a later time, the drawn symbol is shown to the user as an 
available symbol to have a universal command assigned to 
it. 

FIG. 7 shows a preferred embodiment of the creating a 

35 gesture 700. A drawing board is displayed 708. Preferably, 
the drawing board is in a large window. The user 100 will 
place the input device 104 in the location the user wishes to 
start from and then draws the gesture. The coordinates of the 
input device 104 are tracked 712 and dots are displayed at 

40 those coordinates 716, In an embodiment where a mouse is 
being used, the use of the right mouse button can be tracked 
720 to know when the user is signalling that they are 
drawing. However, any button on a mouse could be used, 
and any other input device 104 could be used and tracked to 

45 draw a gesture in accordance with this invention. Gestures 
are composed of line segments which make up the gesture. 
The user draws several lines that will comprise the gesture. 
Each line is shown segmented by handles which can later be 
used for editing the shape of the gesture. The draw a gesture 

50 routine 700 can be used in conjunction with generating a 
universal command or with drawing a symbol independently 
of generating a universal command, as described above. 

After completing the drawing, the user is given an option 
722 to edit the gesture. If the user wishes to edit the gesture 

55 722, then the edit gesture routine 662 is called. To edit a 
gesture, the gesture is displayed with handles marking the 
joining together of the line segments that were used to 
comprise the gesture. FIG. 7c shows a screen shot of an 
editing gesture interface. The user can move the handles to 

60 alter the shape of the figure to whatever shape is desired. 
After editing or if the user chooses not to edit, the gesture is 
analyzed as described above to determine what sequence of 
axes it intersects. After determining the sequence 724, the 
determined sequence is compared 728 to the stored 

65 sequences located in the universal command riles of the 
application to be edited. If this sequence is being used 732, 
then the message is displayed, "Shape in Use, Draw 
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Another" 736 and the user 100 is given another opportunity 
to draw a gesture. If there is no match, then the sequence is 
assigned 740 to the newly created universal command file. 
The system 10 then displays the options of commands or text 
to be assigned to that universal command 612. 5 

FIG. lb displays a preferred embodiment of the create a 
hotkey to represent a universal command routine 668. Here, 
the user selects a hotkey to represent the universal command 
750. The hotkey chosen is compared with the hotkeys stored 
in the universal command files of the universal command 10 
library of the application to be edited 754. If the hotkey is 
being used 758, then the message "Hotkey In Use" 762 is 
displayed and the user 100 is given a chance to select 
another 750. If the hotkey is not being used, then the hotkey 
is assigned to the newly created universal command file 766. ^ 
The system 10 then displays the options of commands or text 
to be assigned to that universal command 612. 

FIG. 8 shows a preferred embodiment of deleting a 
universal command. The user 100 activates the universal 
command editor 802. The user selects the delete command 20 
option 804. The system 10 displays the available applica- 
tions with universal command libraries 806. The user selects 
the library in which the user wants to delete the command 

807. The universal commands of that library are displayed 

808. The user selects a universal command for deletion 812. 25 
If a gesture is chosen, then the universal command file is 
deleted from the universal command library 816. If a hotkey 

is chosen, the central processing unit is instructed to process 
that hotkey if entered by the user 818. 

The universal command generation technology outlined 30 
above is especially useful when applied to second devices, 
such as a television, a CD player, a handheld computer, or 
a stereo system. The apparatus as described in FIG. 1 can be 
embedded onto a single chip 900 as shown in FIG. 9. The 
chip can then be installed onto a remote control that allows 35 
gesture input, or onto the device itself. The user can use the 
user interface 904, which may be a writepad or touchscreen 
to generate a universal command, such as increasing volume 
by drawing a gesture or changing (browsing) a page. The 
user interface is coupled to a recognition module 910. The 40 
recognition module 910 is composed of several components. 
A boundary locating module 911 will locate the boundaries 
of the gesture in accordance with the above description. A 
divider module 912 will divide the bounded gesture into 
regions divided by axes. A mapping module 913 will map 45 
the order in which the gesture intersects the axes. Memory 
914 will store a look up table holding a list of sequences and 
their corresponding commands recognizable by the second 
device. A comparing module 915 will compare the mapped 
input signal sequence to the stored sequences to find a 50 
matching sequence. When a match is found, the output 
module 920 will send the commands assigned to the 
matched sequence to the second device for processing. Thus, 
an increase volume gesture drawn by the user can be 
converted into an increase volume command recognizable 55 
by a second device, for example a television, CD player, or 
a stereo system. All commands recognizable by a television, 
or any second device 140 can thus be represented by 
gestures. Thus, in accordance with the present invention, the 
command conversion technology in conjunction with the 60 
gesture recognition technology can bring the capability of 
user-defined commands and user-friendly gesture input to 
nearly any technological device on the market, without 
necessarily having to make any alteration to the technologi- 
cal device itself. However, the chip can be installed on a new 65 
product to allow a user to have gesture recognition via an 
interface coupled to the device itself. 



Thus, in accordance with the present invention, a user can 
define a single command that will execute any command on 
an application 132, operating system 136, or second device 
140. In accordance with the present invention, libraries of 
universal commands are automatically and dynamically 
loaded responsive to the movement of an input device 104. 
Additionally, in accordance with the present invention, the 
user is able to designate a single command or another event 
to perform the same function on multiple applications, thus 
freeing the user from having to remember specific com- 
mands for specific applications. In accordance with the 
present invention, when a user switches back and forth 
between application, the user can use the same commands 
for both applications. The gesture recognition technology of 
the present invention allows the user to draw the commands 
using a standard input device 104, providing to the user the 
ease of gesture input for all of the user's command needs. 
The gesture recognition technology is highly accurate, using 
dynamic tolerance correction, but does not restrict the user 
to a set of pre-defined characters. The user can create any 
character the user desires to issue a universal command, 
without being limited to pre-defined alphabets. The user can 
store symbols independent of universal commands for later 
use with the universal commands. Additionally, the novel 
command conversion technology of the present invention 
allows gesture recognition capability to be applied to any 
remote device. Thus, any remote device 140 can now be 
controlled by a user-defined gesture, in accordance with the 
present invention. 

While the present invention has been described with 
reference to certain preferred embodiments, those skilled in 
the art will recognize that various modifications may be 
provided. For example, different remote devices may be 
used, different operating systems may be used, or different 
algorithms for drawing the lines may be used consistent with 
the scope of this invention. These and other variations upon 
and modifications to the preferred embodiment provided for 
by the present invention which is limited only by the 
following claims. 

What is claimed is: 

1. In a system having multiple contexts, a method for 
translating signals from an input device into commands 
operating on a context-sensitive basis, wherein at least one 
universal command file is created and stored in a universal 
command library, and at least one universal command 
library is created and associated with at least one of the 
multiple contexts, and each universal command file contains 
a unique sequence and storage space to store at least one 
command, the method comprising: 

determining coordinates of the input device; 
identifying the current context; 

loading the universal command library associated with the 

identified current context; 
recognizing a universal command input transmitted by the 

input device; 

differentiating between an operating system command 
and a universal command during the step of recogniz- 
ing the universal command; 

correlating the recognized universal command input with 
a universal command file stored in the universal com- 
mand library; and 

sending commands stored within the correlated universal 
command file to the current context for execution. 

2. The method of claim 1, wherein the recognizing a 
universal command step further comprises the steps of: 

determining whether a universal command is being trans- 
mitted by the input device; 
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responsive to determining that the universal command is 
being transmitted, drawing a graphical representation 
of the universal command responsive to the input 
signal; 

creating a boundary box around the graphical represen- s 
tation; 

subdividing the boundary box into regions separated by 

multiple axes; and 
determining a sequence of the input signal intersecting the 

axes. 

3. The method of claim 2, wherein subdividing the 
boundary box further comprises the steps of: 

generating a first and a second axes equidistant from the 
top and bottom sides of the boundary box; and 5 

generating a third and a fourth axes equidistant from the 
left and right sides of the boundary box. 

4. The method of claim 2, wherein the correlating step 
comprises the substeps of: 

comparing the determined sequence to a stored sequence 20 
associated with the at least one universal command file 
stored in the loaded universal command library; and 

performing dynamic tolerance until a matching sequence 
corresponding to a universal command is found. 

5. The method of claim 4, wherein performing dynamic 25 
tolerance further comprises the steps of: 

identifying an orientation of the first element in the 

determined sequence; 
identifying an orientation of the next element in the 

determined sequence 30 
swapping the first and second elements of the sequence 

responsive to the first and second sequence elements 

having different orientations to provide a swapped 

sequence; 

comparing the swapped sequence to the stored sequence 35 
associated with at least one universal command file 
stored in the loaded universal command library; 

repeatedly swapping first and second elements of the 
sequence responsive to no match being found or the ^ 
orientations of the first sequence elements being the 
same until all possible permutations have been 
swapped or a match is found. 

6. The method of claim 5, wherein there is limit to the 
number of swaps that can be performed, and step of repeat- 45 
ing comparing the swapped sequence comprises: 

repeatedly comparing the swapped sequence until all 
possible permutations are swapped and compared or a 
match is found or the limit on the number of swaps 
allowed is reached. 50 

7. The method of claim 1, wherein the input device is a 
keyboard, the loaded universal command library contains 
keyboard commands associated with at least one universal 
command file, there is an operating system protocol gov- 
erning the system, and recognizing a universal command 5S 
further comprises the steps of: 

instructing the operating system not to process keyboard 
entries corresponding to the keyboard entries stored in 
the library; and 

comparing the keyboard input to a list of stored keyboard 60 
inputs to find a matching universal command file. 

8. The method of claim 1, wherein sending commands 
further comprises the steps of: 

sending an entry stored in the correlated universal com- 
mand file to the current context for processing; 65 

determining if the sent entry was the last entry in the 
universal command file; and 
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repeatedly sending and determining until all entries in the 
universal command file have been sent. 

9. The method of claim 8 wherein the entry contains text 
strings. 

10. The method of claim 8 wherein one of the contexts is 
an operating system, and an entry contains a operating 
system command. 

11. The method of claim 1, wherein the current context is 
identified based on locating coordinates of the input device. 

12. The method of claim 1 wherein there is an operating 
system coupled to the system, a user can activate one of the 
multiple contexts, and the current context is identified by 
querying the operating system as to what is the active 
context. 

13. A method for creating a new universal command, the 
new universal command is recognized for a specified 
context, and the method comprising the steps of: 

loading a universal command file of the specified context; 

creating a new universal command file; 

displaying a window for drawing a universal command; 

repeatedly drawing a line segment responsive to the input 
signal in response to the input device being configured 
for drawing until the input device is no longer config- 
ured for drawing; 

subdividing the window into regions separated by mul- 
tiple axes; 

determining a sequence of axes intersected by the input 
signal; 

comparing the determined sequence to a sequence corre- 
lating to the at least one universal command file stored 
in the loaded universal command library to find if an 
existing universal command file matches the input 
signal sequence; 

performing dynamic tolerance if no matching sequence is 
found to determine if a matching sequence correspond- 
ing to a universal command file exists; and 

storing the determined sequence into the created universal 
command file if no match exists. 

14. The method of claim 13 wherein options for com- 
mands to be assigned to the new universal command are 
displayed, commands are assigned responsive to the selec- 
tion of an option, and the display is updated responsive to the 
assigning of a command. 

15. A device for interpreting and translating universal 
commands into commands understandable by a second 
device, comprising: 

a recognition module, for receiving an input universal 
command, further comprising: 
a boundary locating module, coupled to input device, 

for locating the boundaries of a transmitted universal 

command; 

a divider module, coupled to the boundary locator, for 
dividing the bounded universal command into 
regions divided by multiple axes; 

a mapping module, coupled to the divider, for gener- 
ating a sequence in which the input signal intersects 
the axes; 

a memory, for storing a list of sequences, each sequence 
associated with one or more commands recognizable 
by the second device; and 

a comparing module, coupled to the memory, for 
comparing the generated sequence to the stored list 
of sequences to find a matching sequence; and 
an output module, coupled to the comparator, for sending 

the commands associated with the matched sequence to 

the second device. 
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16. The device of claim 15, wherein the recognition 
module further comprises: 

a dynamic tolerance adjuster, coupled to the comparator, 
for adjusting the input signal to match a stored 
sequence if the generated sequence does not match a 5 
stored sequence. 

17. The device of claim 15, wherein the modules are 
located on a single device. 

18. The device of claim 15, wherein the first device is a 
remote control and the second device is a device capable of 1° 
being controlled by a remote control. 

19. A method for translating an input signal in the form of 
a gesture into a one or more commands recognizable by a 
computer system comprising: 

receiving an input signal; 15 
creating a boundary box around the input signal; 
subdividing the boundary box into regions separated by 
multiple axes; 

determining a sequence the input signal intersects the 20 
axes; 

comparing the determined sequence to a stored list of 

sequences to find a stored sequence that matches the 

determined sequence; 
performing dynamic tolerance if no matching sequence is 25 

found until a matching sequence is found; and 
sending a command understandable by the second device 

corresponding to the matching stored sequence. 

20. The method of 19 wherein performing dynamic tol- 3Q 
erance further comprises the steps of: 

swapping a first and second elements of the sequence to 
produce a swapped sequence; 

comparing the swapped sequence with the stored list of 
sequences; 35 

repeatedly swapping the first and second elements of the 
sequence to produce at least one additional swapped 
sequence and comparing the additional swapped 
sequence responsive to the no matching sequence being 
found until all permutations have been swapped or a A0 
match is found. 
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21. The method of claim 20, wherein no swaps are made 
if the orientation of the sequence elements to be swapped are 
the same. 

22. The method of claim 20, wherein there is a preset limit 
of swaps, and no swaps are made if the number swaps that 
have already been made exceed the preset limit. 

23. The method of claim 19, wherein the step of deter- 
mining a sequence recognizes straight lines by performing 
the steps of: 

comparing a height and width ratio of the boundary box 
to a first predefined height and width ratio; 

comparing the height and width ratio of the boundary box 
to a second predefined height and width ratio; and 

comparing the sequence the input signal intersects the 
axes to a stored list of sequences defined as straight 
lines to find a stored sequence that matches the deter- 
mined sequence, responsive to the height and width 
ratio of the boundary box being less than the first 
predefined height and width ratio or greater than the 
second predefined height and width ratio. 

24. The method of claim 23, wherein the stored list of 
sequence contains a sequence that is represents a line drawn 
on top of itself, and at least one entry is assigned to said 
sequence, wherein the step of determining a sequence rec- 
ognizes a line drawn on top of itself by performing the steps 
of: 

determining the region a start point of the input signal 

sequence is located within; 
determining the region a end point of the input signal 

sequence is located within; 
comparing the start point region and the end point region 

to determine if the start point and the end point are 

located in the same region; and 
responsive to determining that the start and end point are 

located within the same region, sending the entry 

assigned to the stored sequence that represents a line 

drawn on top of itself. 

* * * * * 
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